Skip to content

ggml-hexagon: fix swiglu failure at test-backend-ops #17344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 26 commits into
base: master
Choose a base branch
from
Open

ggml-hexagon: fix swiglu failure at test-backend-ops #17344

wants to merge 26 commits into from
+99 −45

Conversation

@chraac
Copy link
Contributor

@chraac chraac commented Nov 18, 2025

Fixes some failure on the Hexagon swiglu/silu op and improves correctness of the GGML Hexagon backend.

Summary of changes

  • Add overflow-guarded HVX primitives (hvx_vec_exp_fp32_guard, hvx_vec_inverse_fp32_guard) and use them in the Hexagon backend.
  • Improve NaN/Inf handling in exp, inverse, and related helper func.
  • Fix mistakes in the Hexagon silu / swiglu implementations (including handling of src1 and swapped/split variants).

Before

[SILU] NMSE = 3.457965465 > 0.000000100   SILU(type=f32,ne_a=[128,2,2,2],v=0): �[1;31mFAIL�[0m
[SILU] NMSE = 0.496767445 > 0.000000100   SILU(type=f32,ne_a=[5,7,11,13],v=0): �[1;31mFAIL�[0m
[SWIGLU] NMSE = 3894.832699597 > 0.000000100   SWIGLU(type=f32,ne_a=[128,2,2,2],v=0,swapped=0): �[1;31mFAIL�[0m
[SWIGLU] NMSE = 2.032236263 > 0.000000100   SWIGLU(type=f32,ne_a=[5,7,11,13],v=0,swapped=0): �[1;31mFAIL�[0m
[SWIGLU] NMSE = 531.844516626 > 0.000000100   SWIGLU(type=f32,ne_a=[128,2,2,2],v=0,swapped=1): �[1;31mFAIL�[0m
[SWIGLU] NMSE = 1.988691331 > 0.000000100   SWIGLU(type=f32,ne_a=[5,7,11,13],v=0,swapped=1): �[1;31mFAIL�[0m
[SWIGLU] NMSE = 1040.190893229 > 0.000000100   SWIGLU(type=f32,ne_a=[128,2,2,2],v=0,split): �[1;31mFAIL�[0m
[SWIGLU] NMSE = 0.493175916 > 0.000000100   SWIGLU(type=f32,ne_a=[5,7,11,13],v=0,split): �[1;31mFAIL�[0m

After

[SILU] NaN at index 231 (HTP0=-nan CPU=88.449669)   SILU(type=f32,ne_a=[128,2,2,2],v=0): �[1;31mFAIL�[0m
  SILU(type=f32,ne_a=[5,7,11,13],v=0): �[1;32mOK�[0m
[SWIGLU] NaN at index 122 (HTP0=nan CPU=-11446.431641)   SWIGLU(type=f32,ne_a=[128,2,2,2],v=0,swapped=0): �[1;31mFAIL�[0m
  SWIGLU(type=f32,ne_a=[5,7,11,13],v=0,swapped=0): �[1;32mOK�[0m
[SWIGLU] NMSE = 3.835742624 > 0.000000100   SWIGLU(type=f32,ne_a=[128,2,2,2],v=0,swapped=1): �[1;31mFAIL�[0m
  SWIGLU(type=f32,ne_a=[5,7,11,13],v=0,swapped=1): �[1;32mOK�[0m
[SWIGLU] NaN at index 216 (HTP0=nan CPU=-8444.154297)   SWIGLU(type=f32,ne_a=[128,2,2,2],v=0,split): �[1;31mFAIL�[0m
  SWIGLU(type=f32,ne_a=[5,7,11,13],v=0,split): �[1;32mOK�[0m

chraac added 23 commits November 13, 2025 20:25
@chraac
refactor: use hvx_vec_exp_fp32_guard_inf for overflow handling in hvx…
ab75281 
…_exp_f32
@chraac
feat: add fast sigmoid function with overflow guard for fp32
5aa4a83
@chraac
refactor: replace hvx_vec_inverse_fp32 with hvx_vec_inverse_fp32_guar…
a64154c 
…d_inf for improved overflow handling
@chraac
feat: enhance hvx_add_scalar_f32 with overflow handling using infinit…
a8cdbcf 
…y guard
@chraac
wip
ae42fb6
@chraac
add HVX_Vector_Alias
39445ab 
wip
@chraac
wip
a589b61
@chraac
fix: improve handling of src1 tensor in glu_swiglu_fp32_per_thread fu…
6f57b9e 
…nction
@chraac
fix nc
db9e930
@chraac
wip
ce48af5
@chraac
wip
fc5f31f
@chraac
handle nan at inverse
5707384
@chraac
wip
54235e3
@chraac
fix neg
3859465
@chraac
wip
014ad77
@chraac
rename
8c37457
@chraac
fix hvx_vec_inverse_fp32_guard_inf to handle infinity and NaN cases c…
33a050e 
…orrectly
@chraac
wip
83884a5
@chraac
fix hvx_vec_inverse_fp32_guard_inf to handle NaN cases correctly
f7662f3
@chraac
Merge branch 'master' into dev-fix-swiglu
5f553f0
@chraac
wip
f6d7f3c
@chraac
wip
37e9a1d
@chraac
wip
185dc20
@chraac chraac marked this pull request as draft November 18, 2025 03:24
chraac
chraac commented Nov 18, 2025
View reviewed changes
ggml/src/ggml-hexagon/htp/hvx-exp.c
static const float kMaxExp = 88.02f; // log(INF)

const HVX_Vector max_exp = hvx_vec_splat_fp32(kMaxExp);
const HVX_Vector inf = hvx_vec_splat_fp32(kInf);
Copy link
Contributor Author

@chraac chraac Nov 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought we can move this init out of the for loop below.

chraac
chraac commented Nov 18, 2025
View reviewed changes
ggml/src/ggml-hexagon/htp/hvx-utils.h
uint32_t w[VLEN_FP32];
__fp16 fp16[VLEN_FP16];
float fp32[VLEN_FP32];
} __attribute__((aligned(VLEN), packed)) HVX_VectorAlias;
Copy link
Contributor Author

@chraac chraac Nov 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its safe to use the gcc ext since in htp we're using clang.

chraac
chraac commented Nov 18, 2025
View reviewed changes
ggml/src/ggml-hexagon/htp/act-ops.c
const float limit = ((const float *) (op_params))[3];

const int nc = (src1_valid) ? ne0 : ne0 / 2;
const int nc = (src1_valid) ? ne00 : ne00 / 2;
Copy link
Contributor Author

@chraac chraac Nov 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks we should use src0->ne[0] here instead of dst->ne[0]

chraac
chraac commented Nov 18, 2025
View reviewed changes
ggml/src/ggml-hexagon/htp/hvx-utils.h
// neg by setting the fp32 sign bit
HVX_Vector mask = Q6_V_vsplat_R(0x80000000);
return Q6_V_vor_VV(v, mask);
return Q6_V_vxor_VV(v, mask);
Copy link
Contributor Author

@chraac chraac Nov 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using xor here to fix when x < 0

@DajanaV DajanaV mentioned this pull request Nov 18, 2025
Open
@github-actions github-actions bot added the ggml changes relating to the ggml tensor library for machine learning label Nov 18, 2025
@chraac chraac marked this pull request as ready for review November 19, 2025 01:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@max-krasnyansky max-krasnyansky Awaiting requested review from max-krasnyansky max-krasnyansky is a code owner

@lhez lhez Awaiting requested review from lhez lhez is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@chraac